As so-called “cyber-attacks” (used here as a general term to cover such activities as denial of service (DOS), including Distributed Denial of Service (DDOS), attacks and attempts to infect target computer devices with malicious software—e.g. as part of a DOS attack or simply in order to steal information—e.g. credit card details of customers—etc.) increase in sophistication, they are becoming both more difficult to detect using a single detector and at the same time they are tending to appear more and more like multi-stage attacks, passing though several distinct stages which can be identified (as distinct stages) by skilled human security experts.
Thus, although there are known monitors for detecting known signatures of malicious traffic and/or activities at various different detectors associated with various different typical stages of a multi-stage cyber-attack, it is often difficult to detect a sophisticated multi-stage attack from the use of a single monitor alone (or even multiple different monitors acting in isolation). Instead, such sophisticated multi-stage attacks can often only successfully be detected by linking various different activities (generally detected by different detectors) together and examining them together as aspects of a single multi-stage attack.
For example, DDOS attacks often start with some bad Public relations (PR) of a particular target organization observed on news or social media websites, or with some new vulnerability of a target organization being made public. Then potential attackers may start to talk about the target and to exchange ideas for an attack and recruit or combine forces with other attackers. At some time after that, detection of worms carrying specific DOS payloads may be observed. Then scanning activities may be observed on target networks associated with the target organization, and/or specific HTTP request volumes may increase and compromised/infected machines loaded with DOS malware may be detected. Finally, attacks are launched from multiple machines both within and without the target organizations networks are launched at specific coordinated times in order to bring vital services associated with the target organization down.
As mentioned, such multi-stage attacks can often defeat individual point checks and can only be detected by linking and examining together the various different stages of the attack. For example, login failures are quite common and unlikely to result in a major security incident. However, login failures, followed by a successful login, and obtaining admin rights (by a malicious unauthorized user), and then installing (malicious) software and then observing abnormal traffic flowing over the network is very likely in total to be indicative of a successful attack.
Various approaches for either automatically or semi-automatically identifying attacks by looking for these distinct multiple stages of attack have been proposed and a selection of such proposals is set out below:
“Multi-stage Intrusion Detection System Using Hidden Markov Model (HMM) Algorithm” by Do-hyeon Lee, Doo-young Kim, Jae-il Jung (2008 International Conference on Information Science and Security) proposes a multi-stage Intrusion Detection System (IDS) architecture using an HMM Algorithm and presents a method for determining an intrusion/attack by estimation of the features appearing at each stage of an intrusion. The intrusion techniques used at each stage (e.g. to do network probing) are detected using “characteristic intrusion signals” (i.e. rule sets). Each attack stage has a detection agent which performs an independent detection function. It analyses the data collected from a network line to recognize signals that are known to be intrusions (using the rule sets). The signal sequences detected by the detection agents are then synthesized and the HMM algorithm is used to determine whether the synthesized detected sequences corresponds to an intrusion sequence. The method and system described in this paper aim to produce a better IDS which can correlate “local” alerts (i.e. intrusion signals) at different points in time in order to identify the eventual intrusion/attack objective. Each detection agent (for each stage) appears to be independent of each other detection agent; hence it doesn't have any information from the previous stage regarding certain attack target. Rather, correlation (e.g. if the same destination IP address is used) is performed only when the agents' detection signals are being synthesized. The system doesn't consider, for example, external factors such as discussions in social media, or a new vulnerability being announced.
“Applications of Hidden Markov Models to Detecting Multi-stage Network Attacks” by Ourston et al. (Proceedings of the 36th Hawaii International Conference on System Sciences—2003) describes an approach for detecting multi-stage attacks using Hidden Markov Models (HMMs) and compares the usefulness of the HMM approach to other machine learning techniques. The paper rather glosses over the exact manner in which visible state values are determined, presumably because whatever method is used to achieve this is the same for the different machine learning techniques being compared such that the comparison should still be valid whatever method is chosen. However, they do suggest using multiple detectors and pre-processing the outputs of the detectors, and they suggest using human experts to perform some mapping between alert types (e.g. a port probe) and intrusion category (e.g. initial recon). This mapping though appears to be between an observable value and a hidden state and so does not make clear the nature of the observable values presented to the HMM model. In any event, the main conclusion of the paper is that HMM's are well suited to the task of detecting multi-stage network attacks.
“Automatic attack scenario discovering based on a new alert correlation method” by Ali Ebrahimi et al (Systems Conference (SYSCON), 2011 IEEE International, IEEE, 4 Apr. 2011, pages 52-58) teaches a method of identifying a multi-stage attack. The method includes collecting multiple Intrusion Detection System (IDS) alerts and grouping them into a single group based on source port and within that overall group, sub-grouping the alerts, based on target IP address, into different sub-groups, and then attempting to allocate the alerts within each sub-group to further sub-groups which correspond to different stages of a multi-stage attack.
“Addressing Low Base Rates in Intrusion Detection via Uncertainty-Bounding Multi-Step Analysis” by Robert Cole and Peng Liu (Computer Security Applications Conference 2008 (ACSAC 2008) Annual, IEEE, Piscataway, N.J., USA, 8 Dec. 2008 pages 269-278) describes a system in which a user is given control to specify a trade-off between confidence that an attack is occurring (confidence) and how quickly an attack is detected (agility). The authors recognize that in general there will always be a trade-off between these two characteristics within a multi-stage attack detection system and that by giving control over where the balance should lie to a user they can tailor the system to their own needs (e.g. if a user has large human resources for analyzing possible attacks and a strong desire to catch attacks as quickly as possible they would tend to choose for a more agile system, whereas user's that can only afford relatively small amount of resource for investigating possible attacks would tend to prefer to receive fewer false positives and thus would tend to opt for a slower but more reliable system (with few false positives). In terms of a system for actually detecting multi-stage events, this paper described only a very simple system in which it was assumed that each detection by a sensor could be unambiguously associated with a particular stage of a particular multi-stage attack with a simple probability of being right or wrong. This is sufficient for the purposes of demonstrating their trade-off issues but, as was acknowledged by the authors, would not be useful for creating a practical multi-stage attack detection system which would thus need to borrow heavily from known prior art systems for correlating sensor outputs with multi-stage attacks.
“An analysis Approach for Multi-stage Network Attacks” describes a method of analyzing a system which is vulnerable to attack using a Multi-stage Finite State Machine Model. Using such a model, network administrators can identify optimum ways of improving the security of the system to prevent the occurrence of successful attacks. There is no teaching of detecting currently occurring multi-stage attacks, and so it does not describe a multi-stage event detector.
A difficulty faced by all systems attempting to detect multi-stage events occurring in relatively complex systems where more than one multi-stage event may be occurring within overlapping periods of time, is how to efficiently keep track of an individual multi-stage event occurring and not getting obfuscated by the detection of overlapping multi-stage events, and additionally how to deal efficiently with the possibility that in its early stages a multi-stage event may be indistinguishable (or difficult to distinguish) from other events with similar or identical starting behaviours, especially if different attacks are occurring from the same attacker/source of attack.